Rust database access via Python database connection pool v2 by MadLittleMods · Pull Request #19878 · element-hq/synapse

MadLittleMods · 2026-06-24T23:54:55Z

Based on the explorations done in #19824 and #19846,

Rust database access via Python database connection pool

This is a stepping stone before we can go full Rust everywhere. We're providing a generic interface as we want database access to work in Synapse and synapse-rust-apps. In synapse-rust-apps, we will use a tokio-postgres based database connection pool so it's full Rust.

We want to avoid the situation where we have two database connection pools (one for Python, one for Rust) as we've run into connection exhaustion problems on Matrix.org before.

As an example of using it and sanity check for all this work (including tests), I've also ported over the /versions handler to the Rust side with database access. The /versions endpoint is the simplest endpoint I could find that still had some database access. Hopefully the refactor on /versions isn't that controversial as it's not really the point of this PR. We can always remove it from this PR but it's just here as a sanity check that all of this works.

Why `runInteraction(...)`?

Using the same runInteraction pattern that we already have in Synapse means we can port over existing Synapse code/endpoints without much thought. But this pattern also makes sense because we want¹ transactions to have repeatable-read isolation (easy to think about, less foot-guns). Having everything thappen in a function callback means we can do retries for serialization/deadlock errors.

When an application receives this error message, it should abort the current transaction and retry the whole transaction from the beginning. The second time through, the transaction will see the previously-committed change as part of its initial view of the database, so there is no logical conflict in using the new version of the row as the starting point for the new transaction's update.

Note that only updating transactions might need to be retried; read-only transactions will never have serialization conflicts.

-- https://www.postgresql.org/docs/current/transaction-iso.html#XACT-REPEATABLE-READ

As a note, this strategy is less of an impedance mismatch (aligns more closely) with Synapse so the glue code for the python_db_pool should also be simpler.

How does this interact with logcontext (`LoggingContext`)?

See docs on log contexts for more background.

We already support normal logging from Rust -> Python with pyo3-log and log but as soon as we pass a thread boundary, everything is logged against the sentinel log context. Normally, we want logs and CPU/DB usage correlated with the request that spawned the work.

You can see how I took a stab at fixing this in #19846 by capturing the logcontext in a Tokio task local and re-activating as necessary. For example, in that PR, I reactivated the logcontext in run_python_awaitable(...) which we use to call runInteraction(...) from the Rust side which means all of the database usage is correlated with the request as expected. It also means any log:info!(...) done in run_interaction(...) is correlated correctly. But there needs to be a better story for when you want to log everywhere else.

I haven't explored tracking CPU usage on the Rust side.

I've left all of this out of this PR as I think it will be better to tackle this as a dedicated follow-up. For example, I'm thinking about instead creating a new LoggingContext with the parent_context set to the calling context and try to avoid needing to call set_current_context(...) on the Python side where possible (like tracking CPU).

Testing strategy

Added some tests that exercise some async Rust handlers for the /versions endpoint:

SYNAPSE_TEST_LOG_LEVEL=INFO poetry run trial tests.rest.client.test_versions.VersionsTestCase

Real-world:

poetry run synapse_homeserver --config-path homeserver.yaml
GET http://localhost:8008/_matrix/client/versions

Dev notes

Example INSERT

self.db_pool
    .run_interaction("asdf", move |txn| {
        async move {
            let _rows = txn
                .query(
                    "INSERT INTO delayed_events (delay_id, user_localpart, delay, send_ts, room_id, event_type, content, is_processed) \
                     VALUES (?, ?, ?, ?, ?, ?, ?, ?)",
                    &["delay_id", "user_localpart", "10", "999991782342348349", "!room", "m.room.message", "content", "0"],
                )
                .await?;

            Ok(())
        }
        .boxed()
    })
    .await?;

Todo

Audit panic usage (unwrap, expect, panic!, unreachable!)
Wait for Update FakeChannel.await_result(...) to drive async Rust (Tokio runtime/thread pool) #19879

Pull Request Checklist

Pull request is based on the develop branch
Pull request includes a changelog file. The entry should:
- Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
- Use markdown where necessary, mostly for code blocks.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry.
Code style is correct (run the linters)

To note: Ideally, we'd want the least isolation possible but the problem is that there is no tooling to yell at you when your queries/logic is wrong so repeatable-read isolation is a great balance. ↩

`/versions` is a good candidate as its is the simplest endpoint that uses the database. We use `tests/rest/client/test_versions.py` as a sanity check that all of this new Rust database access works.

MadLittleMods · 2026-06-25T00:09:19Z

@@ -0,0 +1 @@
+Allow Rust code to have database access via Python database connection pool.


I suggest reviewing this commit by commit.

It first introduces the database interface structure, then Store usage, and then finally refactoring the /versions endpoint to be handled in Rust.

The /versions endpoint is the simplest endpoint I could find that still had some database access. Hopefully the refactor on /versions isn't that controversial as it's not really the point of this PR. We can always remove it from this PR but it's just here as a sanity check that all of this works.

I've looked over and refined all of this but feel free to call out whatever as I might be cargo-culting too much from the LLM splat that this is based on, #19846 (that PR is a combination of LLM splats and manual refinement)

MadLittleMods · 2026-06-25T00:12:00Z

+        # XXX: We must create the Rust HTTP client before we call `reactor.run()` below.
+        # Twisted's `MemoryReactor` doesn't invoke `callWhenRunning` callbacks if it's
+        # already running and we rely on that to start the Tokio thread pool in Rust. In
+        # the future, this may not matter, see https://github.com/twisted/twisted/pull/12514


As an update, twisted/twisted#12514 is finally part of a Twisted release 26.4.0 (2026-05-11). If we updated our Twisted version, we could probably get rid of all of this ugliness.

But that may be a few years away given our deprecation policy considers a "no-brainer" upgrade once it's available in both the latest Debian Stable (currently Twisted 24.11.0) and Ubuntu LTS repositories (currently Twisted 25.5.0)

MadLittleMods · 2026-06-25T00:12:33Z

+    // We can't check this here because of circular import issues
+    // logging_context_module(py)?;


Perhaps this will get fixed by #19876 (comment)

MadLittleMods · 2026-06-25T00:13:29Z

                RoomCreationPreset.TRUSTED_PRIVATE_CHAT,
                RoomCreationPreset.PUBLIC_CHAT,
-            ]
+            }


Made these a set to better represent what it is and more efficient lookups. Touched this because I had to model it in SynapseHomeServerConfig on the Rust side

MadLittleMods · 2026-06-25T00:15:22Z

+
+/// Experimental features the server supports
+#[derive(Serialize, Debug, Clone)]
+pub struct UnstableFeatureMap {


For reference, this is a manual translation from synapse/rest/client/versions.py#L105-L213 (not a prompt and pray it's right). I also already asked an LLM to review and spot any mistakes.

MadLittleMods · 2026-06-25T00:17:39Z

@@ -0,0 +1,343 @@
+/*


For reference, you can also see what a Rust tokio-postgres based DatabasePool looks like in #19846

It's completely unscrutinized but the point is just that it's possible and we can care about it once we start using it in synapse-rust-apps.

MadLittleMods · 2026-06-25T00:19:51Z

+                // We fire-and-forget using `run_in_background`. Re-using
+                // `run_in_background` also makes sure the awaitable gets run with the
+                // current logcontext while following the logcontext rules.
+                //
+                // FIXME: Currently runs in the sentinel logcontext because we don't manage it here


You can see how handling logcontext might look like in #19846 but per my explanation in the PR description here, it's being left out in favor of a follow-up PR.

MadLittleMods · 2026-06-25T00:23:32Z

+logger = logging.getLogger(__name__)
+
+
+class VersionsTestCase(unittest.HomeserverTestCase):


I'm relying on these tests to sanity check that the whole Rust database access thing works.

Is this good enough? Should I be writing tests on the Rust side? Seems hard/weird as our only DatabasePool implementation requires Synapse to be running. How do we make that happen?

I wish I could write some specific Rust db_pool.run_interaction(...)/txn.query(...) scenarios and test results better but generally I guess this is good enough. We could expose that function and run it from here 🤷

MadLittleMods · 2026-06-25T00:24:39Z

    def transport(self) -> "FakeChannel":
        return self

    def await_result(self, timeout_ms: int = 1000) -> None:


These changes are based on the same decisions being made in #19871

This seems to cause our trial tests not to finish in CI 🤔

Even tried these changes by themselves in #19879 which also reproduces.

I'll have to figure it out tomorrow but I think this PR is still good to review otherwise.

Split off to #19879

Note the tests will fail here until we figure out #19879

MadLittleMods · 2026-06-25T00:38:07Z

+struct RustHandlers {
+    versions: Py<versions::VersionsHandler>,
+}


Opinions on self.rust_handlers.versions vs exposing handlers individually?

If this is the pattern we like, we should also move rust/src/rendezvous / rust/src/msc4388_rendezvous here as well (follow-up PR).

MadLittleMods · 2026-06-25T00:39:39Z

+
+#[pyclass]
+pub struct VersionsHandler {
+    pub global_unstable_feature_map: Arc<UnstableFeatureMap>,


We have the UnstableFeatureMap abstraction separate from SynapseHomeServerConfig as we potentially want to re-use these handlers as part of synapse-rust-apps, Rusty homeserver, etc. We don't want to bog it down with Synapse specific things.

MadLittleMods · 2026-06-25T00:42:06Z

+            "v1.10".to_string(),
+            "v1.11".to_string(),
+            "v1.12".to_string(),
+        ]),


To make this actually generic (usable outside of Synapse), we probably want to pass this in as well but I've left it for now ⏩

MadLittleMods · 2026-06-25T16:21:57Z

+    // ```
+    // db_pool.run_interaction("description", async move |txn| {
+    //     /* do stuff with txn */
+    // })
+    // ```


This kind of thing may be possible. It seems like sea-orm accomplished this somehow:

Feature Request: Support for async closures SeaQL/sea-orm#2749 (comment)

https://blog.weiznich.de/blog/diesel-async-0-9/

In any case, we can address this in a follow-up PR.

See #19878 (comment)

So we can stop muddying up the actual config

See #19878 (comment)

…-db-pool2

MadLittleMods · 2026-07-03T22:47:05Z

+    /// Checks whether a given feature is enabled/disabled for this user
+    ///
+    /// If there is no entry, returns None
+    pub async fn is_feature_enabled_for_user(


One thing that is different from the Python is_feature_enabled(...) is that we don't have any cached lookups on the Rust version

synapse/synapse/storage/databases/main/experimental_features.py

Lines 101 to 104 in 807795e

@cached()

async def is_feature_enabled(

self, user_id: str, feature: "ExperimentalFeature"

) -> bool:

I think this fine for now, it's quite a small table so lookups should be cheap.

erikjohnston · 2026-07-06T13:00:07Z

@MadLittleMods cargo test seems to have broken ftr

…-db-pool2

erikjohnston

Other than the test failures I think this looks good

See https://doc.rust-lang.org/rustdoc/write-documentation/documentation-tests.html#attributes > The `ignore` attribute tells Rust to ignore your code > The `no_run` attribute will compile your code but not run it.

…time/thread pool) (#19879) Split off from #19878 (needed to drive requests in tests that use async Rust) Follow-up to #19871

…-db-pool2 Conflicts: tests/server.py

``` [FAIL] Traceback (most recent call last): File "/home/runner/work/synapse/synapse/tests/push/test_http.py", line 1037, in test_msc3881_client_versions_flag self.assertEqual(channel.code, 200) File "/home/runner/.cache/pypoetry/virtualenvs/matrix-synapse-pswDeSvb-py3.14/lib/python3.14/site-packages/twisted/trial/_synctest.py", line 444, in assertEqual super().assertEqual(first, second, msg) File "/opt/hostedtoolcache/Python/3.14.6/x64/lib/python3.14/unittest/case.py", line 925, in assertEqual assertion_func(first, second, msg=msg) File "/opt/hostedtoolcache/Python/3.14.6/x64/lib/python3.14/unittest/case.py", line 918, in _baseAssertEqual raise self.failureException(msg) twisted.trial.unittest.FailTest: 500 != 200 tests.push.test_http.HTTPPusherTests.test_msc3881_client_versions_flag =============================================================================== [FAIL] Traceback (most recent call last): File "/home/runner/work/synapse/synapse/tests/rest/client/test_delayed_events.py", line 45, in test_false_by_default self.assertEqual(channel.code, 200, channel.result) File "/home/runner/.cache/pypoetry/virtualenvs/matrix-synapse-pswDeSvb-py3.14/lib/python3.14/site-packages/twisted/trial/_synctest.py", line 444, in assertEqual super().assertEqual(first, second, msg) File "/opt/hostedtoolcache/Python/3.14.6/x64/lib/python3.14/unittest/case.py", line 925, in assertEqual assertion_func(first, second, msg=msg) File "/opt/hostedtoolcache/Python/3.14.6/x64/lib/python3.14/unittest/case.py", line 918, in _baseAssertEqual raise self.failureException(msg) twisted.trial.unittest.FailTest: 500 != 200 : {'version': b'1.1', 'code': b'500', 'reason': b'Internal Server Error', 'headers': Headers({b'Server': [b'1'], b'Date': [b'Tue, 07 Jul 2026 09:15:50 GMT'], b'Cache-Control': [b'public, max-age=600, s-maxage=3600, stale-while-revalidate=600'], b'Vary': [b'Authorization'], b'Content-Type': [b'application/json'], b'Access-Control-Allow-Origin': [b'*'], b'Access-Control-Allow-Methods': [b'GET, HEAD, POST, PUT, DELETE, OPTIONS'], b'Access-Control-Allow-Headers': [b'X-Requested-With, Content-Type, Authorization, Date'], b'Access-Control-Expose-Headers': [b'Synapse-Trace-Id, Server']}), 'body': b'{"errcode":"M_UNKNOWN","error":"Internal server error"}', 'done': True} tests.rest.client.test_delayed_events.DelayedEventsUnstableSupportTestCase.test_false_by_default =============================================================================== [FAIL] Traceback (most recent call last): File "/home/runner/work/synapse/synapse/tests/rest/client/test_delayed_events.py", line 51, in test_true_if_enabled self.assertEqual(channel.code, 200, channel.result) File "/home/runner/.cache/pypoetry/virtualenvs/matrix-synapse-pswDeSvb-py3.14/lib/python3.14/site-packages/twisted/trial/_synctest.py", line 444, in assertEqual super().assertEqual(first, second, msg) File "/opt/hostedtoolcache/Python/3.14.6/x64/lib/python3.14/unittest/case.py", line 925, in assertEqual assertion_func(first, second, msg=msg) File "/opt/hostedtoolcache/Python/3.14.6/x64/lib/python3.14/unittest/case.py", line 918, in _baseAssertEqual raise self.failureException(msg) twisted.trial.unittest.FailTest: 500 != 200 : {'version': b'1.1', 'code': b'500', 'reason': b'Internal Server Error', 'headers': Headers({b'Server': [b'1'], b'Date': [b'Tue, 07 Jul 2026 09:15:50 GMT'], b'Cache-Control': [b'public, max-age=600, s-maxage=3600, stale-while-revalidate=600'], b'Vary': [b'Authorization'], b'Content-Type': [b'application/json'], b'Access-Control-Allow-Origin': [b'*'], b'Access-Control-Allow-Methods': [b'GET, HEAD, POST, PUT, DELETE, OPTIONS'], b'Access-Control-Allow-Headers': [b'X-Requested-With, Content-Type, Authorization, Date'], b'Access-Control-Expose-Headers': [b'Synapse-Trace-Id, Server']}), 'body': b'{"errcode":"M_UNKNOWN","error":"Internal server error"}', 'done': True} tests.rest.client.test_delayed_events.DelayedEventsUnstableSupportTestCase.test_true_if_enabled =============================================================================== [FAIL] Traceback (most recent call last): File "/home/runner/work/synapse/synapse/tests/rest/client/test_login_token_request.py", line 159, in test_unstable_support self.assertEqual(channel.code, 200) File "/home/runner/.cache/pypoetry/virtualenvs/matrix-synapse-pswDeSvb-py3.14/lib/python3.14/site-packages/twisted/trial/_synctest.py", line 444, in assertEqual super().assertEqual(first, second, msg) File "/opt/hostedtoolcache/Python/3.14.6/x64/lib/python3.14/unittest/case.py", line 925, in assertEqual assertion_func(first, second, msg=msg) File "/opt/hostedtoolcache/Python/3.14.6/x64/lib/python3.14/unittest/case.py", line 918, in _baseAssertEqual raise self.failureException(msg) twisted.trial.unittest.FailTest: 500 != 200 tests.rest.client.test_login_token_request.LoginTokenRequestServletTestCase.test_unstable_support =============================================================================== [FAIL] Traceback (most recent call last): File "/home/runner/work/synapse/synapse/tests/rest/client/test_matrixrtc.py", line 116, in test_msc4143_false_by_default self.assertEqual(channel.code, 200, channel.result) File "/home/runner/.cache/pypoetry/virtualenvs/matrix-synapse-pswDeSvb-py3.14/lib/python3.14/site-packages/twisted/trial/_synctest.py", line 444, in assertEqual super().assertEqual(first, second, msg) File "/opt/hostedtoolcache/Python/3.14.6/x64/lib/python3.14/unittest/case.py", line 925, in assertEqual assertion_func(first, second, msg=msg) File "/opt/hostedtoolcache/Python/3.14.6/x64/lib/python3.14/unittest/case.py", line 918, in _baseAssertEqual raise self.failureException(msg) twisted.trial.unittest.FailTest: 500 != 200 : {'version': b'1.1', 'code': b'500', 'reason': b'Internal Server Error', 'headers': Headers({b'Server': [b'1'], b'Date': [b'Tue, 07 Jul 2026 09:15:57 GMT'], b'Cache-Control': [b'public, max-age=600, s-maxage=3600, stale-while-revalidate=600'], b'Vary': [b'Authorization'], b'Content-Type': [b'application/json'], b'Access-Control-Allow-Origin': [b'*'], b'Access-Control-Allow-Methods': [b'GET, HEAD, POST, PUT, DELETE, OPTIONS'], b'Access-Control-Allow-Headers': [b'X-Requested-With, Content-Type, Authorization, Date'], b'Access-Control-Expose-Headers': [b'Synapse-Trace-Id, Server']}), 'body': b'{"errcode":"M_UNKNOWN","error":"Internal server error"}', 'done': True} tests.rest.client.test_matrixrtc.MatrixRtcVersionsTestCase.test_msc4143_false_by_default =============================================================================== [FAIL] Traceback (most recent call last): File "/home/runner/work/synapse/synapse/tests/rest/client/test_matrixrtc.py", line 122, in test_msc4143_true_if_enabled self.assertEqual(channel.code, 200, channel.result) File "/home/runner/.cache/pypoetry/virtualenvs/matrix-synapse-pswDeSvb-py3.14/lib/python3.14/site-packages/twisted/trial/_synctest.py", line 444, in assertEqual super().assertEqual(first, second, msg) File "/opt/hostedtoolcache/Python/3.14.6/x64/lib/python3.14/unittest/case.py", line 925, in assertEqual assertion_func(first, second, msg=msg) File "/opt/hostedtoolcache/Python/3.14.6/x64/lib/python3.14/unittest/case.py", line 918, in _baseAssertEqual raise self.failureException(msg) twisted.trial.unittest.FailTest: 500 != 200 : {'version': b'1.1', 'code': b'500', 'reason': b'Internal Server Error', 'headers': Headers({b'Server': [b'1'], b'Date': [b'Tue, 07 Jul 2026 09:15:57 GMT'], b'Cache-Control': [b'public, max-age=600, s-maxage=3600, stale-while-revalidate=600'], b'Vary': [b'Authorization'], b'Content-Type': [b'application/json'], b'Access-Control-Allow-Origin': [b'*'], b'Access-Control-Allow-Methods': [b'GET, HEAD, POST, PUT, DELETE, OPTIONS'], b'Access-Control-Allow-Headers': [b'X-Requested-With, Content-Type, Authorization, Date'], b'Access-Control-Expose-Headers': [b'Synapse-Trace-Id, Server']}), 'body': b'{"errcode":"M_UNKNOWN","error":"Internal server error"}', 'done': True} tests.rest.client.test_matrixrtc.MatrixRtcVersionsTestCase.test_msc4143_true_if_enabled ```

MadLittleMods · 2026-07-07T09:39:54Z

+        # XXX: We must create the Rust HTTP client before we call `reactor.run()` below.
+        # Twisted's `MemoryReactor` doesn't invoke `callWhenRunning` callbacks if it's
+        # already running and we rely on that to start the Tokio thread pool in Rust. In
+        # the future, this may not matter, see https://github.com/twisted/twisted/pull/12514
+        self._http_client = hs.get_proxied_http_client()
+        _ = HttpClient(
+            reactor=hs.get_reactor(),
+            user_agent=self._http_client.user_agent.decode("utf8"),
+        )


Had to spread this to a few other places that deal with /versions since it uses async Rust now 😢

But this is our standard pattern for dealing with starting the Tokio runtime.

See #19878 (comment) for more details on how this could go away.

Given we already subclass for lots of tests here:

synapse/tests/server.py

Line 585 in 026937b

class ThreadedMemoryReactorClock(MemoryReactorClock):

I wonder if we can patch it for now by overriding callWhenRunning to automatically run stuff if the reactor is already running?

I think it's possible. We can consider it in a follow-up PR ⏩

erikjohnston

Looks good! Whoop!

MadLittleMods · 2026-07-07T18:10:04Z

Thanks for the review @erikjohnston and @reivilibre for the good tips and references on a previous exploratory stab at things 🐘

MadLittleMods added 5 commits June 24, 2026 18:29

Add Rust database interface

9714a94

Add run_python_awaitable(...)

6da263b

Add Python implementation of the DatabasePool

cee06f9

Add Store with example usage of run_interaction(...)

9225c8e

Update /versions endpoint to use Rust

d978a66

`/versions` is a good candidate as its is the simplest endpoint that uses the database. We use `tests/rest/client/test_versions.py` as a sanity check that all of this new Rust database access works.

MadLittleMods added rust Z-Rust labels Jun 24, 2026

Add changelog

35163d8